Skip to content

fix(pages-router): stream piped API responses with backpressure - #2735

Open
NathanDrake2406 wants to merge 5 commits into
cloudflare:mainfrom
NathanDrake2406:fix/pages-api-stream-backpressure
Open

fix(pages-router): stream piped API responses with backpressure#2735
NathanDrake2406 wants to merge 5 commits into
cloudflare:mainfrom
NathanDrake2406:fix/pages-api-stream-backpressure

Conversation

@NathanDrake2406

Copy link
Copy Markdown
Contributor

Overview

Goal Make piped Pages API responses (stream proxying, #1902) deliver incrementally with bounded memory
Core change The response bridge holds write callbacks while the body's queue is full, and the Node production server streams live API bodies instead of buffering them
Key boundary PagesResponseStream owns backpressure; handlePagesApiRoute owns the stream-vs-complete decision; adapters only consume it
Expected impact A piped source pauses when the client reads slowly; proxied SSE/long-lived streams flush incrementally on Node prod. res.json/res.send/res.end(data) behaviour is unchanged

Why

Node streams propagate backpressure by withholding the _write callback: a held callback makes write() return false, which pauses any pipe() source. PagesResponseStream._write enqueued each chunk into the Web ReadableStream and invoked the callback synchronously, so the Writable looked infinitely fast. A fast source piped into res (the proxy pattern #1902 enabled) kept producing at full rate while a slow client consumed nothing, and every unread chunk accumulated in the stream's unbounded queue. On real Next.js this cannot happen because res is a genuine ServerResponse with socket backpressure; the bridge silently dropped that property.

Independently, the Node production server read API responses with arrayBuffer() before sending. That predates streaming API bodies and was harmless while every API response was complete at resolution time. Once a response can resolve mid-stream, buffering holds the whole body in memory and defers the first byte until the source closes, so a proxied SSE stream never flushes.

Area Principle / invariant What this PR changes
PagesResponseStream A Writable must not acknowledge writes faster than its consumer drains Write callback is parked while desiredSize <= 0 and released from the stream's pull() hook (or on destroy)
handlePagesApiRoute Only the bridge layer can know whether a body is complete or live Responses still being written when the handler settles are marked streamed
Node prod server A live stream must be forwarded, not materialised Marked responses take the existing sendWebResponse streaming path, with the same application/octet-stream content-type fallback the buffered path applied

What changed

Scenario Before After
upstream.pipe(res) with a slow client Source drained at full rate; unread chunks queued without bound Source pauses after roughly one queued chunk plus the Writable's 16 KB buffer, resumes per consumer read
Piped/streaming API response on Node prod Fully buffered via arrayBuffer(); first byte only after the source closed Streams through sendWebResponse; bytes flush as they are written
res.json / res.send / res.end(data) Buffered send with Content-Length Unchanged (complete bodies are not marked streamed and keep the buffered path)
Client cancels mid-stream Writable destroyed Unchanged, and a parked write callback is released so piped sources unwind
Middleware headers merged onto an API response Marker n/a Streamed marker survives the mergeHeaders rewrap (same propagation as the streamed-HTML marker)
Maintainer review path
  1. packages/vinext/src/server/pages-node-compat.ts for the backpressure mechanism (_write parking, pull() release, _destroy release) and the marker helpers.
  2. packages/vinext/src/server/pages-api-route.ts for the stream-vs-complete decision after the response settles.
  3. packages/vinext/src/server/prod-server.ts for the stream-vs-buffer branch and content-type fallback.
  4. packages/vinext/src/server/pages-request-pipeline.ts and packages/vinext/src/shims/unified-request-context.ts for marker propagation across Response rewraps.
  5. tests/pages-api-route.test.ts for the regression proof.
Validation
  • New regression test: a 20-chunk (64 KB each) piped source must not drain while nothing reads the body, and completes fully once the body is consumed.
  • New contract test: a response settled mid-pipe is marked streamed; res.json is not.
  • All 70 tests in tests/pages-api-route.test.ts and tests/pages-bodyparser-config.test.ts pass, including the existing fix(pages-router): support stream proxying in API routes #1902 streaming, cancellation, and destroy-mid-stream cases.
  • tests/pages-node-compat.test.ts, tests/pages-request-pipeline.test.ts, tests/unified-request-context.test.ts pass (118 tests).
  • vp check clean on all touched files; pre-commit full check, staged tests, and knip passed.
Risk / compatibility
  • No public API change; the marker is a non-enumerable-style internal property consumed only by vinext adapters.
  • Complete-body responses keep the buffered path, so Content-Length emission on Node prod (fix(metadata): preserve Content-Length for fully buffered responses #2703 behaviour) is unchanged.
  • Responses that stream then finish quickly now go chunked on Node prod instead of buffered with Content-Length. This matches Next.js, where res.write() implies chunked transfer.
  • A handler that writes and never ends nor pipes still relies on the existing auto-end/cancellation paths; parked callbacks are always released on destroy, so no new hang path is introduced.

References

Reference Why it matters
#1902 Introduced stream proxying for Pages API routes; this PR makes that pattern flow-controlled and actually incremental on Node prod
#2703 Content-Length preservation for buffered responses, deliberately kept intact for complete bodies

A handler that pipes into `res` (proxying an upstream, echoing a request
body) could queue the entire stream in memory. The response bridge
acknowledged every write immediately, so `write()` never returned false
and a piped source was never paused, regardless of how slowly the client
consumed the body. The Node production server also read API responses
with `arrayBuffer()`, which held the full body in memory and deferred
delivery until the source closed, so long-lived streams (e.g. proxied
SSE) never flushed.

Hold the write callback while the response body's queue is full and
release it from the stream's pull hook. A held callback makes `write()`
return false, which pauses any piped source until the consumer reads.

Mark responses that are still being written when the handler settles and
send them through the streaming path in the Node production server.
Complete bodies (`res.json`, `res.send`, `res.end(data)`) keep the
buffered path and its Content-Length behaviour.
@pkg-pr-new

pkg-pr-new Bot commented Jul 27, 2026

Copy link
Copy Markdown

Open in StackBlitz

npm i https://pkg.pr.new/@vinext/cloudflare@2735
npm i https://pkg.pr.new/create-vinext-app@2735
npm i https://pkg.pr.new/@vinext/types@2735
npm i https://pkg.pr.new/vinext@2735

commit: 598c370

@github-actions

github-actions Bot commented Jul 27, 2026

Copy link
Copy Markdown
Contributor

Performance benchmarks

Compared 598c370 against base 25dc2f3 using alternating same-runner rounds. Next.js was unchanged and skipped.

0 improved · 0 regressed · 6 within ±1.5%

Scenario Framework Baseline Current Change
Client bundle size (gzip) vinext 134.3 KB 134.3 KB ⚫ -0.0%
Client entry size (gzip) vinext 121.9 KB 121.9 KB ⚫ +0.0%
Dev server cold start vinext 2.89 s 2.87 s ⚫ -0.7%
Production build time vinext 3.02 s 2.98 s ⚫ -1.1%
RSC entry closure size (gzip) vinext 111.3 KB 111.3 KB ⚫ +0.0%
Server bundle size (gzip) vinext 189.0 KB 189.0 KB ⚫ +0.0%

View detailed results and traces

🟢 improvement · 🔴 regression · ⚫ change below 1.5% · paired base/head

@james-elicx

Copy link
Copy Markdown
Member

/bigbonk review for issues

@ask-bonk ask-bonk Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the backpressure mechanism, the stream-vs-complete decision, the prod-server branch, and marker propagation. The core design is sound and well-tested: I verified the _write parking + pull() release cycle works for serialized writes, that buffered responses (res.json, res.send, large res.end(data)) still complete because the reading adapter pulls and releases the parked callback, that no unhandled rejection leaks when a streaming handler rejects without a ctx, and that dev/prod parity now holds (the dev middleware already streamed via Readable.fromWeb().pipe(res); prod's arrayBuffer() buffering was the outlier this fixes). Cloudflare Workers streams API bodies natively so it inherits backpressure without needing the marker. Nice work.

One correctness-of-documentation issue and two smaller notes below. Nothing blocking.

Verification performed:

  • Simulated the desiredSize <= 0 park / pull() release cycle — releases correctly on read, and desiredSize ?? 1 avoids deadlock on closed/errored controllers.
  • Confirmed res.json / large res.end(data) / write()+end() are not marked streamed and complete fully.
  • Confirmed the reject-after-write and cancel-parked-write paths settle the parked callback with the error.
  • Ran tests/pages-api-route.test.ts, tests/pages-bodyparser-config.test.ts, tests/after-deploy.test.ts, tests/pages-request-pipeline.test.ts, tests/unified-request-context.test.ts, tests/pages-node-compat.test.ts — all green.

if (
this.controller &&
!this.streamEnded &&
!this.writableEnded &&

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The comment claims end(data) writes are finished immediately, but the !this.writableEnded guard doesn't actually achieve that. When res.end(data) runs, Node calls _write synchronously before end() sets writableEnded, so inside this _write this.writableEnded === false. I confirmed this empirically:

_write writableEnded=false   <- the end(data) write
_final
after end() writableEnded=true

And end-to-end: a large res.end(bigData) parks its write callback and the end() completion callback does not fire until the body is read (endCallbackFired before read: false). So finish is gated on the first read for a large end(data), exactly what the comment says it avoids.

This is functionally harmless today — the buffering adapter reads the body via arrayBuffer(), which pulls and releases the parked callback, and the response is correctly not marked streamed (because finalizeResponse checks writableEnded, which is true after end() returns). But the stated rationale is inaccurate and could mislead a future maintainer who relies on end(data) writes not parking.

If you actually want end(data) writes to bypass parking, gate on writableEnding (set synchronously at the start of end(), unlike writableEnded) instead of writableEnded:

Suggested change
!this.writableEnded &&
if (
this.controller &&
!this.streamEnded &&
!this.writableEnding &&
(this.controller.desiredSize ?? 1) <= 0
) {

Otherwise, please reword the comment to drop the "finish them immediately" claim, since parking-then-releasing-on-read is the real behavior.

) {
response.headers.set("content-type", result.defaultContentType);
}
await sendWebResponse(response, req, res, compress);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor behavioral note (not blocking): routing streamed API responses through sendWebResponse means compressible content-types now compress in "streaming" flush mode regardless of the 1024-byte COMPRESS_THRESHOLD that the buffered sendCompressed path applies. For a small streamed application/json/text/plain body this adds gzip/br overhead where the buffered path would have skipped it. SSE (text/event-stream) is unaffected since it's not in COMPRESSIBLE_TYPES. Probably fine — a live stream has no known length anyway — but worth a one-line comment so the divergence from the buffered threshold is intentional and documented.

if (merged !== response) {
(merged as { __vinextStreamedApiResponse?: boolean }).__vinextStreamedApiResponse = (
response as { __vinextStreamedApiResponse?: boolean }
).__vinextStreamedApiResponse;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For consistency: the Cloudflare mergeHeaders in server/worker-utils.ts does not copy __vinextStreamedApiResponse (it only strips content-length for the HTML marker). This is not a bug — the Workers path returns new Response(response.body, ...) and streams natively, so it doesn't consume the API marker — but the asymmetry (prod-server + this pipeline propagate it, worker-utils drops it) is a latent trap if a future change makes the Workers adapter branch on the marker. Consider a brief comment in worker-utils.mergeHeaders noting the API marker is intentionally not propagated because Workers streams unconditionally.

@ask-bonk

ask-bonk Bot commented Jul 29, 2026

Copy link
Copy Markdown
Contributor

Review posted successfully. Here's a summary of my review of PR #2735.

Review Summary

The PR fixes a real availability problem: piped Pages API responses (upstream.pipe(res), the proxy pattern from #1902) drained a fast source at full rate into an unbounded ReadableStream queue while a slow client read nothing, and the Node prod server buffered the whole body via arrayBuffer() before sending. The design is sound and well-tested.

What I verified (all green):

  • The _write parking + pull() release cycle works for serialized Writable writes and doesn't deadlock (desiredSize ?? 1 guards closed/errored controllers).
  • Buffered responses (res.json, res.send, large res.end(data), write()+end()) are correctly not marked streamed and complete fully because the reading adapter pulls and releases parked callbacks.
  • Reject-after-write and cancel-parked-write paths settle the parked callback with the error; no unhandled rejection leaks when a streaming handler rejects without a ctx.
  • Dev/prod parity: the dev middleware already streamed (Readable.fromWeb().pipe(res)), prod's buffering was the outlier this fixes. Cloudflare Workers streams natively.
  • Ran the 6 relevant test suites.

Findings (none blocking):

  1. Inaccurate comment in pages-node-compat.ts — the !this.writableEnded guard does not exempt end(data) writes from parking, because Node calls _write synchronously before end() sets writableEnded. I proved empirically that a large res.end(bigData) parks and its completion callback doesn't fire until the body is read. Harmless (adapter reads → pulls → releases; response not mis-marked), but I suggested either gating on writableEnding or reworded the comment.

  2. Minor compression divergence — streamed API responses bypass the buffered path's 1024-byte COMPRESS_THRESHOLD. SSE unaffected; suggested a documenting comment.

  3. Marker asymmetry — Cloudflare's worker-utils.mergeHeaders doesn't propagate __vinextStreamedApiResponse (not a bug since Workers streams unconditionally), but suggested a note to avoid a future trap.

github run

@james-elicx

Copy link
Copy Markdown
Member

/bigbonk review for issues

@ask-bonk ask-bonk Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the backpressure mechanism, the stream-vs-complete decision in handlePagesApiRoute, the prod-server stream branch, and marker propagation. The core design is sound and the change correctly maps Node socket backpressure onto the Fetch ReadableStream boundary. I verified:

  • Parking + release cycle: _write parks the callback only once this.controller is set (which happens synchronously via resolveOnce() on the first write) and desiredSize <= 0. Since Writable serializes _write, there is at most one pendingWrite at a time, so releasing one per pull() (and on _destroy) is correct and cannot leak. desiredSize ?? 1 avoids deadlock on a closed/errored controller.
  • Buffered bodies still complete: res.json / res.send / large res.end(data) are not marked streamed (checked after writableEnded is true) and the buffered adapter's arrayBuffer() read pulls and releases any parked terminal write.
  • Error paths: handler-throws-after-write errors the stream via res.destroy and reports once through handlerLifecycle; handler-throws-before-write still yields a 500 via the outer catch, and the losing race branch is handled (no unhandled rejection).
  • Workers lifecycle: the streaming branch returns before the handler settles, so registering handlerLifecycle on the active execution context is the right fix; the new after-deploy.test.ts covers the ambient/hybrid case.
  • Ran tests/pages-api-route.test.ts, tests/after-deploy.test.ts, tests/pages-node-compat.test.ts, tests/pages-request-pipeline.test.ts — all green.

Contrary to the earlier bot review, the pages-node-compat.ts comment is accurate: it explicitly states end(data) writes can park, which matches the code. Two small non-blocking notes below.

if (
this.controller &&
!this.streamEnded &&
!this.writableEnded &&

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Edge case worth a note: a parked write is only released by a consumer pull() or by _destroy. If the returned Response body is never read and never cancelled, the final parked write stays parked, _final never runs, and any handler awaiting pipeline()/drain never settles (its promise is only kept alive via waitUntil). This mirrors real Node (an unconsumed socket also stalls the producer), so it's acceptable — but since this is the one path that can leave a handler promise pending indefinitely, a one-line comment here documenting that release depends on consumption would help future maintainers reason about it.

) {
response.headers.set("content-type", result.defaultContentType);
}
await sendWebResponse(response, req, res, compress);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor behavioral divergence (not blocking): routing streamed API responses through sendWebResponse means a compressible content-type (application/json, text/plain) now compresses regardless of the buffered path's 1024-byte COMPRESS_THRESHOLD. A small streamed JSON body will gzip/br where the buffered path would have skipped it. text/event-stream is unaffected (not in COMPRESSIBLE_TYPES). A live stream has no known length so this is defensible, but a one-line comment noting the threshold intentionally doesn't apply to streamed bodies would make the divergence explicit.

if (merged !== response) {
(merged as { __vinextStreamedApiResponse?: boolean }).__vinextStreamedApiResponse = (
response as { __vinextStreamedApiResponse?: boolean }
).__vinextStreamedApiResponse;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maintainability: __vinextStreamedApiResponse is now a magic string duplicated across pages-api-route.ts, pages-node-compat.ts (typed helpers), this file (inline cast), unified-request-context.ts (inline cast), and prod-server.ts (inline reader). pages-node-compat.ts already exports markVinextStreamedApiResponse / isVinextStreamedApiResponse; consider reusing those helpers (or a shared const for the key) at these copy sites so a future rename can't silently miss one. Mirrors the existing __vinextStreamedHtmlResponse pattern, so this is a pre-existing style choice rather than a new problem.

@ask-bonk

ask-bonk Bot commented Jul 29, 2026

Copy link
Copy Markdown
Contributor

Review posted successfully.

Summary

I reviewed PR #2735 (streaming piped Pages API responses with backpressure) and posted a batch review with 3 non-blocking inline comments.

Verdict: The core design is sound and correct. The change properly maps Node socket backpressure onto the Fetch ReadableStream boundary.

What I verified (all green):

  • Parking/release cycle_write parks the callback only once the controller exists and desiredSize <= 0; Writable serialization guarantees at most one pendingWrite, so one release per pull()/_destroy cannot leak. desiredSize ?? 1 avoids deadlock on closed/errored controllers.
  • Buffered bodies (res.json/res.send/res.end(data)) are correctly not marked streamed and still complete because the adapter's arrayBuffer() read releases the parked terminal write.
  • Error paths — throw-after-write errors the stream and reports once; throw-before-write still yields a 500; no unhandled rejection in the losing race branch.
  • Workers lifecycle — registering handlerLifecycle on the active execution context is the right fix since the streaming branch returns before the handler settles; the new after-deploy.test.ts covers the hybrid case.
  • Ran 4 relevant test suites — all pass.

Non-blocking findings:

  1. Parked writes only release on consumption/destroy — an unread+uncancelled body can leave a handler promise pending (matches real Node; worth a doc comment).
  2. Streamed API responses bypass the buffered 1024-byte compression threshold (defensible; worth a comment).
  3. __vinextStreamedApiResponse magic string is duplicated across 5 sites while typed helpers already exist (maintainability).

I also noted that, contrary to the earlier bot review, the pages-node-compat.ts comment is actually accurate about end(data) writes being able to park.

github run

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants